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process  -  perhaps  a  ’wideband’  noise  driven  system  or  a  suitably  scaled  discrete 
parameter  process.  The  optimal  controls  for  these  ’physical’  processes  are 
usually  nearly  impossible  to  obtain.  Thus,  it  is  of  considerable  interest  to  know 
whether,  the  optimal  (or  8-optimal  control  for  the  diffusion  model  is  ’nearly’ 
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convergence  analysis  via  the  Skorohod  topology  on  D[0,  JR.  not  appropriate 


here,  due  to  the  nature  of  the  singular  controls.  A  combination  of  the 
Skorohod  and  ’pseudopath’  topology  is  adapted  to  our  singular  control  problem 
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Abstract 


Singular  control  problems  with  diffusion  or  Wiener  process  systems  have 
been  occuring  with  increasing  frequency  as  models  of  a  wide  variety  of 
applications;  e.g.,  storage,  inventory,  finite  fuel,  consumption  and  investment, 
limits  of  impulsive  control  problems,  etc.  Here,  the  increment  of  the  control 
force  is  not  of  the  usual  form  u(t)dt,  but  is  the  differential  of  a 
non-decreasing  and  suitably  adapted  process.  The  models  used  (Wiener  or 
diffusion  processes)  are  only  approximations  in  some  sense  to  some  ’physical* 
process  -  perhaps  a  ’wideband’  noise  driven  system  or  a  suitably  scaled  discrete 
parameter  process.  The  optimal  controls  for  these  ’physical’  processes  are 
usually  nearly  impossible  to  obtain.  Thus,  it  is  of  considerable  interest  to  know 
whether  the  optimal  (or  6-optimal  control  for  the  diffusion  model  is  ’nearly’ 
optimum  when  applied  to  the  physical  problem,  when  compared  to  the  optimal 
or  6-optimal  control  for  the  latter  problem.  This  is  true,  under  broad 
conditions.  The  discounted  and  average  cost  per  unit  time  problems  are  treated. 
The  main  methods  are  those  of  weak  convergence  theory.  But  the  usual  weak 
convergence  analysis  via  the  Skorohod  topology  on  D[0,*)  is  not  appropriate 
here,  due  to  the  nature  of  the  singular  controls.  A  combination  of  the 
Skorohod  and  ’pseudopath’  topology  is  adapted  to  our  singular  control  problem 
to  give  the  required  results. 


Key  Words:  singular  stochastic  control,  approximately  optimal  control,  weak 
convergence,  pscudopath  topology,  control  of  wideband  noisy 
systems,  modelling  of  physical  systems  by  singular  control  processes. 


1.  Introduction 


Let  Yj(-)  ,  i  -  0,1,  be  non-decreasing  processes  with  Y;(0)  -  0  and  which 
are  non-anticipative  with  respect  to  a  Wiener  process  w(-)  .  Define  x(-),  Y(-) 
and  Z(  )  by  x(0)  =  x,  Z(0)  -0  ,  Y(-)  »  Y0()  -  Yx(-)  and  (which  defines  Z()) 

(1.1)  dx  =  b(x)  dt  +  o(x)  dw  +  dY0  -  dYx  =  dZ  +  dY  . 

We  assume  that  there  is  a  B  €  (0,*)  such  that  we  are  obliged  to  keep  x(t)  € 
[0,B].  Unless  otherwise  mentioned,  we  always  assume  that  the  Y(  )  process  is 
such  that  x(t)  €  [0,B]  .  The  process  (1.1)  has  been  widely  used  as  a  model 
of  storage  and  dam  processes,  both  with  and  without  control  [1]  -  [5].  The 
Yj(  )  might  denote  the  ’withdrawal’  process,  whereby  actual  use  is  made  of  the 
system’s  contents.  Y0()  might  simply  denote  a  process  which  is  used  solely  as 
a  modelling  device  to  guarantee  that  x(t)  *  0  (in  that  case,  dY0(t)  «  0  if 
x(t)  *  0).  Sec,  e.g.  [4],  The  process  Z(-)  might  denote  the  difference  between 
the  ’natural’  inputs  and  ’natural’  demand.  See,  in  particular,  the  discussion  in 
[4]  on  this  point.  Many  other  interpretations  are  possible  and  are  discussed  in 
the  references. 

Let  k0  >  kj  >  0  and  let  k(  )  be  a  bounded  continuous  function  Define 
the  two  types  of  costs  (Ex  denotes  the  expectation  under  initial  condition  x(0)  *  x) 

V0(x,Y0,Y1)  -  E,  J  e-0t  |k0dY0(t)  -  kjdYJt)  +  k(x(t))  dt]  , 

(1.2)  o'-  J 

V(x)  -  inf  V0(x,Y0,Y1) 

_ _  T 

70(x,Y0,Y1)  -  lim  Ex  J  [k0dY0(t)  -  ^dYjO)  +  k(x(t))dt]/T 
T  o 

70(x)  -  inf  y0(x,Y0,Y1)  . 

Yo,Yi 


(1.3) 


,  V  f,** 


In  (2.1),  the  inf  is  over  all  non-anticipative  Yj(-)  such  that  x(t)  €  [0,B].  The  class 
over  which  the  inf  is  taken  in  (1.3)  will  be  described  in  Section  7. 

Reference  [1]  gives  an  elegant  presentation  of  the  optimal  control  problem 

(1.1) ,  (1.2),  and  of  the  properties  of  the  associated  Bellman  equation.  Most  of 
the  other  current  literature  seems  to  concern  the  case  where  Z()  is  a  Wiener 
process  (perhaps  with  drift).  Since  it  is  highly  unlikely  that  there  are  problems 
in  the  applications  which  are  perfectly  modelled  by  (1.1)  or  where  Z(  )  is 
actually  a  Wiener  process,  one  must  look  at  the  model  in  the  sense  that  it 
approximates  in  some  way  an  actual  physical  problem.  Generally,  the  model 

(1.1)  would  be  simpler  than  the  ‘physical*  process  which  it  approximates.  In  this 
sense,  it  is  an  attractive  object  to  use  for  purposes  of  calculating  a  control  for 
use  on  the  actual  physical  process.  But  the  ‘quality’  of  such  a  control  when 
used  on  the  physical  process  is  far  from  clear.  E.g.,  how  good  is  it  in 
comparison  with  the  optimal  control  for  the  physical  process. 

Models  such  as  (1.1)  often  arise  as  limits  of  suitably  interpolated  discrete 
parameter  processes.  Consider  one  type:  For  each  €  >  0,  let  (i  =  0,1)  Yf(  )  be 
non-decreasing  processes,  piecewise  constant  on  the  intervals  [ne,  ne+e),  and  define 
8Y.€(n)  h  Y^ne+e)  -  Y^ne).  For  appropriate  functions  F  and  G,  define  (X*,  Z*}  by 
Xq  =  x,  Z*  =  0,  and 


Xn-M  =  Xn  +  <Zn+l  *  Zn>  +  8Y0  (n<>  ‘  8Yl(n€) 

<14>  Z*+l  -  Zn  "  *G(X‘,  (<)  +  Vi  F(X*,  O 

EF(x,  (')50, 

where  {$*}  is  some  correlated  sequence  of  random  variable.  We  say  that  the 
controls  8Yj€(ne)  in  (1.4)  at  time  n  are  admissible  if  they  depend  on  the  ‘full 
information’  (X*,  i  S  n,  Y*(ei),  j  =  0,1,  i  <  n,  f*,  j  <  n)  available  at  time  n.  Define 
the  interpolated  processes  X€(t)  =  X*  and  Z€(t)  =  Z^  on  [n«,  nc  +  t)  and  the  costs 


3 


1 1 


(using  admissible  controls) 

00 

Vq(x,  Y«  Yj)  =  E<  J  e'0t  [k0dY«(t)  -  kjdY^t)  +  k(X€(t))dt] 
o 

(1.5) 

Ve(x)  =  inf  Vg  (x,  Y«  Y<)  . 

Y€Y€ 

Io»I  i 


|  In  general,  we  know  virtually  nothing  about  the  optimal  or  8-optimal  policies  for 

(1.4),  (1.5).  Suppose  that,  for  reasonable  Yf(-),  the  set  (X€(-),  Y^(),  Y*(  )} 
converges  weakly  in  some  sense  to  a  solution  of  (1.1).  It  is  of  considerable  interest 
}  to  know  just  how  good  (compared  with  the  optimal  controls  for  (1.4))  are  the 

,  optimal  (or  6-optimal)  policies  which  we  obtain  for  (1.1),  (1.2),  when  suitably 

adapted  to  and  applied  to  the  system  (1.4),  (1.5).  Consider  the  following  example. 

1  Let  Yj(-),  i=0,I,  denote  the  optimal  or  (for  some  given  6  >  0)  6-optimal 

controls  for  (1.1),  (1.2).  Frequently  [I],  [4],  they  are  of  the  barrier  form:  there 
are  0  <  L*  <  U*  <  *  such  that  Y0(  )  is  used  only  to  keep  x(-)  from  ‘falling 

i  below’  L*  and  Yx(.)  is  used  only  to  keep  x(-)  from  going  above  U*.  The  policy 

Yj(-)  adapted  to  (1.4)  (call  it  Yj€(-))  is  a  policy  which  returns  Xe(  )  to  L*  or  to  U* 
immediately  if  it  ever  drops  below  L*  or  exceeds  U*. 

From  the  point  of  view  of  optimal  control,  one  wants  to  show  that  the  costs 
[  Y€(x)  and  V€(x,  Yq,  Y^)  are  close  for  small  €,  whether  or  not  the  optimal 

controls  for  (1.1),  (1.2)  are  of  the  barrier  policy  type.  It  is  such  a  result  which 

| 

i  justifies  the  use  of  the  limit  model  (1.1)  to  get  policies  which  might  be  used  in 

applications.  Similar  comments  apply  to  the  cost  function  (1.3),  and  also  to  the 
continuous  parameter  analogs  of  (1.4).  This  is  the  class  of  problems  dealt  with 
here.  The  basic  tools  are  those  of  weak  convergence  theory.  The  work  here 

extends  that  in  [6],  [7],  The  results  in  these  references  cannot  be  used  here. 


Owing  to  the  generality  of  the  assumptions  on  the  physical  system,  there  is  in 
general  nothing  known  about  the  properties  of  the  optimal  Y£().  Because  of 
this,  we  cannot  (in  general)  prove  tightness  or  weak  convergence  of 
(X*( •)»  Yq(  ),  YJ()}  in  the  Skorohod  topology  on  D^O,®).  E.g.,  the  jumps  in 
the  Yq,  Y£  might  be  quite  ’wild’— relative  to  the  Skorohod  topology.  A 
weaker  topology  must  be  used,  and  it  is  described  in  Section  2.  In  Section  3, 
we  obtain  a  weak  convergence  result  for  a  continuous  time  model,  and  we  show 
how  to  extend  the  results  to  the  discrete  time  case  in  Section  4.  Section  5 
contains  some  auxiliary  results  which  are  needed  later.  The  optimal  control 
problem  for  the  discounted  cost  case  is  dealt  with  in  Section  6,  where  it  is 
shown  that  the  suitably  adapted  policy  for  the  limit  is  indeed  ’nearly’  optimal 
for  the  actual  physical  process,  under  appropriate  and  reasonable  assumptions. 
Section  7  concerns  the  average  cost  per  unit  time  problem.  Here,  owing  to  the 
natural  requirement  of  stationarity,  we  impose  a  Markov  structure  on  the 
problem. 

The  basic  methods  work  just  as  well  for  many  non-scalar  models—and 
several  extensions  to  such  models  are  discussed  in  Section  8.  Applications  to 
queueing  and  manufacturing  networks  require  a  somewhat  special  development, 
which  will  be  published  separately.  There  are  extensions  of  the  results  to 
cases  where  the  dynamical  terms  are  not  smooth  or  the  noise  is  state  dependent. 
One  would  then  adapt  the  weak  convergence  technique  and  assumptions  of  [8, 
Chapters  5.3,  5.5  and  5.8]  to  the  problem  here.  The  methods  are  quite  similar, 
but  the  assumptions  are  more  complicated.  Our  discussion  is  confined  to  certain 
singular  control  problems.  But  it  is  also  possible  to  work  with  controls  which 
have  both  a  singular  and  an  ‘absolutely  continuous’  component.  One  simply 
combines  the  methods  developed  here  --  with  those  used  in  [6]  for  the 
non-singular  case.  The  extensions  should  be  straightforward.  Possible  extensions 


in  another  direction  concern  the  problem  of  [9],  where  the  singular  control 
problem  appears  as  a  limit  of  impulsive  control  problems:  Let  6£  denote  the 
fixed  cost  per  impulse  and  (for  some  k  >  0)  k|Y£(t)  -  Y€(t*)|  the  variable  cost, 
for  an  impulse  at  time  t.  Then  we  can  let  6€  ■*  0  as  the  noise  bandwidth  goes 
to  *  ,  to  get  an  approximation  theorem  for  small  fixed  impulsive  cost  and  wide 
bandwidth  simultaneously.  The  method  is  a  simple  adaptation  of  the  scheme  of 
this  paper. 


In  this  section,  we  discuss  the  topology  on  D[0,®)  (replacing  the  Skorohod 
topology)  which  will  allow  us  to  obtain  the  desired  weak  convergence  results. 

Let  the  physical  system  be  modelled  by 

X€(t)  =  Z€(t)  +  Y€(t) 

where  X€(0)  =  x,  Z€(0)  =  0,  Y€(-)  =  Y®(->  -  Y£()  and  define 

*(■)  *  [x€( • ),  Y£(),  Y£()  Z€(  )].  The  X*(  ■)  can  be  viewed  either  as  a 
continuous  parameter  interpolation  of  a  discrete  parameter  system  as  discussed 

in  Section  1,  or  it  might  be  an  actual  physical  model  for  a  continuous 
parameter  system.  The  Y.€()  can  be  taken  to  be  either  left  or  right 

continuous,  but  we  suppose  that  they  (and  Xe()  and  Z£(-))  are  right 
continuous. 

We  suppose  that  the  X?(  )  take  values  in  D4[0,®)  ,  the  space  of 
Revalued  functions  which  are  right  continuous  and  have  left  hand  limits.  [We 
could  let  the  Y£()  take  values  in  a  space  of  measures  or  of  distribution 

functions,  but  this  is  actually  not  more  helpful.]  The  appropriate  topology  for 

our  purposes  on  D4[0,®)  is  what  Meyer  [10]  and  Meyer  and  Zheng  [11]  call  the 
pseiidopath  topology.  For  completeness,  we  state  some  definitions  and  results  from 
[11]  which  will  be  needed  in  the  sequel.  The  results  are  stated  for  a  scalar 

valued  process,  but  the  natural  extensions  for  the  Revalued  case  should  be 
obvious  and  are  used  below. 

Let  y(-)  €  D[0,®)  and  define  the  measure  X()  on  the  Borel  subsets  of 
[0,®)  by  X(dt)  ■  e^dt.  Let  P  denote  the  compact  space  of  probability  measures 
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probability  measure  on  the  Borel  subsets  of  [0,*]  x  R  which  is  the  image  of 
V(  )  under  the  map  t  — *  (t,y(t))  of  [0,®]  into  [0,®]  x  R  (i.e.,  it  is  a  point 


in  P).  Let  ip  denote  the  map  which  takes  y(  )  into  its  pseudo-path,  the 
corresponding  point  in  P.  If  we  write  P  =  <Ky(-)),  then  the  pseudopath  P  is  the 
measure  defined  by  (A  and  B  are  Borel  sets  in  [0,®]  and  R,  resp.) 
P(A  x  B)  =  /Ae'^y^gBjdt. 

<p  is  1:1  on  D[0,®),  since  it  identifies  all  paths  which  are  equal  a.e. 
(Lebesque  measure).  The  topology  which  P  induces  on  D[0,®)  via  ip  is  called 
the  pseudo-path  topology.  The  associated  o-algebra  on  D[0,®)  is  the  same  as  one 
gets  with  the  Skorohod  topology.  In  fact  ([11],  Lemma  1  and  comment  after  its 
proof),  the  pseudo-path  topology  on  D[0,®)  is  the  topology  of  convergence  in  measure. 
The  same  result  holds  if  R  and  D[0,®)  are  replaced  by  Rr  and  Dr[0,®), 
where  ip  then  maps  points  y(-)  €  Dr[0,®)  into  a  measure  on  the  Borel  subsets  of 
[0,®]  x  Rr.  Let  Fr  denote  the  space  of  probability  measures  on  the  Borel  subsets  of 
[0,“]  x  Rr. 

The  process  X€(  )  induces  a  measure  (which  we  denote  by  P£)  on  P4  via 
the  pseudo-path  mapping  ip.  The  set  (P£)  is  obviously  tight  since  P4  is 
compact.  If  P  is  a  limit  measure  of  any  weakly  convergent  subsequence,  then  for 


the  convergence  to  be  useful  we  need  at  least  that  P  be  supported  by  <KD4[0,®)), 
since  then  the  limit  P  would  correspond  to  some  process  X()  = 
[x(  • ),  Y0(  •),  YjO),  Z(  •)}  with  paths  in  D4[0,®)  ,  via  the  mapping  ip.  A 
convenient  criterion  for  this  is  given  by  Meyer  and  Zheng  [11],  and  will  now  be 


described. 


Let  T  denote  a  finite  partition  {ti(  i  S  n)  :  0  =  to<ti<-<tn  =  *  •  Let  U(-) 
denote  a  process  with  paths  in  D[0,®)  and  adapted  to  a  non-decreasing  sequence 
of  o-algcbras  {FJ,  and  with  E|U(t)|  <  ®  for  each  t<  ®  .  For  convenience  in 
comparing  with  [11],  let  U(t)  =  0  for  large  t  .  Define  the  variations 


I 


VV»\v'.  %Vw\>\v\v\V1 
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VarT  (U>  -  1  E|Er  U(tj+1)  -  U(t,)  | 

(2.1)  ,<n  ‘i 

Var  (U)  -  Sup  VarT  (U)  . 

If  Var  (U)  <  •»  ,  then  U(  )  is  said  to  be  a  quasimartingale. 

For  u<v  ,  let  NU,V(U)  denote  the  number  of  upcrossings  of  U(  )  on 
[0,*)  between  the  levels  u  and  v.  If  U(  )  is  a  quasimartingale,  then 

I  u  1  +  Var  (U) 

(2.2)  ENUV(U)  <  — -  , 

v  -  u 

an  extension  of  the  usual  result  for  martingales  ([11],  Lemma  3).  The  main 
result  is  ([11],  Theorem  4). 

Theorem  2.1.  For  each  n  =  1,2,  ...,  let  Pn  be  a  probability  law  on  the  Borel *  subsets  of 
D[0,°°)  with  the  associated  process  Un(-)  being  a  quasimartingale  with  sup  Var(Un)  <  °°. 

n 

Then  there  is  a  subsequence  Pn  which  converges  weakly  on  D[0,“)  (with  the  pseudo-path 
topology )  to  a  law  P,  and  the  associated  process  U(-)  is  a  quasimartingale. 
[ Alternatively ,  let  Pn  be  the  measure  induced  on  P  by  the  map  ip  acting  on  Un().  Then 
{Pn}  is  tight  on  P  and  there  is  a  weakly  convergent  subsequence  {Pn^}  with  limit  denoted 
by  P.  P  is  supported  on  D[0,®)  and  the  associated  process  is  a  quasimartingale .] 

Combining  this  with  the  previous  results,  we  have: 

Theorem  2.2.  Assume  the  conditions  and  terminology  of  Theorem  2.1  and  let  h(  )  be 
any  bounded  real  valued  function  on  D[0,<x>)  which  is  continuous  (w.p.l  with  respect  to 
P)  when  the  topology  of  convergence  in  measure  is  used  on  D[0,®).  Then  there  is  a 
subsequence  (nk)  such  that  Eh(Un^(  ))  -*  Eh(U(  )).  Also  ([11],  Theorem  5)  there  is  a 

•The  Skorohod  topology  and  the  pseudopath  topology  generate  the  same  o-algebra 
on  D[0,®). 
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further  subsequence  {mk}  C  {nk}  and  a  set  I  of  full  measure  ( depending  on  P)  such  that 
the  finite  dimensional  distributions  of  {Um  (t),  t  €  I)  converge  to  those  of  (U(t),  t  €  I}. 
Let  f ( - )  be  bounded  and  continuous  on  [0,*).  Then  ([11],  Theorem  6)  the  function 
(tj,  ....  tq)  -•  Ef(U  (tj),  ....  Un^(tq))  converges  in  measure  to  the  function 
<tt . tq)  -  Ef(U(t1),  U(tq)). 
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3.  The  Quasimartingale  Property  and  Weak  Convergence  of 
X€(  )  -  (X€(  •  ),  Z£(),  Y£(.),  Y£()) 


A  continuous  parameter  case  will  be  done  in  this  section.  The  discrete 


parameter  case  requires  only  a  few  modifications  and  is  discussed  in  the  next 


section.  For  concreteness,  we  use  a  specific  model  which  is  of  a  widely  used 


form  [8],  [12],  [13],  for  representing  wide  band  width  noise  driven  systems  (with 
or  without  the  driving  Y  £  ( - )  process).  The  techniques  are  usable  for  a  much 
broader  class  of  systems--_iust  as  for  the  case  where  Y.€(  )  =  0  dealt  with  in  [6] 


or  the  various  continuous  parameter  models  in  [8].  The  model  to  be  used  is 


dX£  =  G(X£,  ?£)dt  +  F(X£,(£  )dt/«  +  dY€(t)  , 


where  t£(t)  =  ?(t/«2)  and  ?(•)  is  a  right  continuous  random  process. 


Let  Ef  denote  the  expectation  conditioned  on  {X£(s),  ?£(s),  Y£(s),  Y£(s), 
s  <  t  }  ,  and  Et  the  expectation  conditioned  on  U(s),  s  «  t)  .  Define 


Z£(t)  5  [  G(X€(s),  i£(s))ds  +  -  f  F(X£(s),  t£(s))ds  . 
J0  £  *0 


VVe  will  use  the  following  assumptions.  Various  extensions  (vector  case. 


discontinuous  dynamics,  state  dependent  noise)  are  possible,  as  discussed  in  the 


introduction,  via  the  appropriate  extension  of  the  methods  in  [8]  for  these  cases 


to  the  problem  at  hand. 


A3.1.  G(-,  ),  F(-,  ),  and  Fx(-,)  are  bounded  continuous  functions  and  the 


latter  two  are  continuous  in  x-  uniformly  in  $. 


A3.2.  For  each  scalar  x,  EF(x,  S(s))  =  0  and  ((•)  is  right  continuous 


and  sufficiently  mixing  such  that  there  is  a  K  <  »  for  which  for  each  T  <  • 


T 

Sup  I  f  Et  g(  x,  *(s))ds  I  <  K  , 

x,t«T  Jt 


where  g(-,  )  represents  either  F(-,  )  or  F  (■,•)  . 
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Theorcm  3.1.  Assume  (A3.1)  and  (A3.2)  and  let  sup[EY0e(t)  +  E  Y  £(t)]  <  • 
for  each  t.  Then  ( with  the  addition  of  a  process  whose  maximum  value  goes  to  zero  as 
t  -»  0),  (X£(),  Y|(-),  Yj(-).  Z£(-)}  are  quasimartingales  with  uniformly  (in  e)  bounded 
variation  on  each  bounded  time  interval.  [)Ve  need  not  assume  that  X€(t)  €  [0,B]  i’«  this 
theorem.] 

Proof.  Since  the  mean  variations  of  the  Y.€(-)  are  bounded,  they  are 
obviously  quasimartingales  with  uniformly  (in  «)  bounded  variation  on  each 
interval  [0,t].  Thus,  we  need  only  work  with  the  Z£(-).  We  will  use  the 
so-called  perturbed  test  function  method  {8],  [12],  [14]  but  adapted  to  our  present 
needs.  For  some  arbitrary— but  large--T,  define  the  process  Z£()  for  t  <  T,  by 

T/€2 

Z£(t)  =  7  }  EfF(X€(t),  l€(s))ds  =  e  |  E£  F(  X£(t),  «s))ds 

t/£J 

The  change  of  variable  s/c2  — *  s  will  be  used  frequently,  in  the  averaging  and 

bounding  in  the  sequel,  when  working  with  integrals  such  as  zj().  By  (A3.2) 

Sup  |  Z£ (t)  |  «  0(£)  . 
t  «  T 

We  will  show  that  the  function  defined  by  f£(t)  «*  Z€ (t)  +  Z£(t)  is  a 
quasimartingale  with  uniformly  (in  «)  bounded  variation  on  each  interval  [0,T]. 
The  calculations  will  be  done  in  a  slightly  indirect  way  so  that  they  can  be 
re-used  later.  Let  f(  )  denote  a  ’test*  function  with  bounded  and  continuous 
derivatives  up  to  order  three,  and  define  f€(t)  -  f(Z£(t))  +  fj(t),  where  f£(  ) 
is  the  ’perturbation’  defined  by 

1  fT 

ff  (t)  -  -  j  fg(Z£(t))E£  F(X£(t),  (£(s))ds 
fT/€2 

=  €  f,(Z€(t))E£  F(X£(t),  ((s))ds. 
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Write  Y.£c(),  Y.£d()  and  Y£()  ,  Yd()  for  the  continuous  and  jump 
components  of  Y£()  and  Y£()  resp.  By  our  convention  on  the  right  continuity  of 
the  Y.£(),  we  use  dY£d(u)  =  Y£d(u)  -  Y£d(u‘).  By  integrating  the  derivative  of 
f(Z€()), 

Ef  f(Z£(t  +  A))  -  f(Z£(t))  = 

(3.3)  *fA  r  eel 

E£  j  f,(  Z£(u))  |G(X£(u),  S£(u))  +  F(X  (u),  j  (u))|ri„ 

t 

Similarly,  by  evaluating  [E£f £  (u+A)  -  f£(u)]/A  and  letting  A  -  0,  we  get 

t+A 

Ef  f l  (t  +  A)  -  f£(t)  -  |  f ,  (  Z£(u))  Ej  F(X£(u),  5e(u))du 

t 

t+A  T 

+  |  du  J-  Et£  |  ds  fM(  Z€(u))  E£  F(X£(u),J£(s))  . 


(3.4) 


F(  Xe(u),  ?e(u)) 


+  G(  X£(u),  l£(u)) 


t+A 


+  E£  J  dYc€(u)  }  J  ff(Z£(u))E£  Fx(Xe(u),  ?£(s))ds 


t+A  r 

I  E> 


F(X£(u),  4£(u))  +  G(X£(u),  5€(u)) 


du 

X 


T 

•  f  E£  f(  Z«(u))  F(  X£(u),  t£(s))ds  + 

e 

u 

T 

+  )  7  Et  I  f,(Z€(u))E£  [f(X£(u")  +  dY£(u), 

it  ^ 


(S)) 


t<u<t+A 


•  F(X!(iT),  (‘(s))lds 
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Recall  that  dY(u)  -  Y(u)  -  Y(u"). 

By  a  change  of  scale  s/«2  — *  s  and  the  use  of  (A3. 1 )  and  (A3.2),  the  second 
and  fourth  terms  on  the  r.h.s.  of  (3.4)  are  seen  to  be  O(A)  .  By  a  similar  scale 
change,  the  third  term  is  seen  to  be  0(«)E*  (Y£(t  +  s)  -  Y£(t)).  The  first  term 
of  (3.4)  is  the  negative  of  the  ’  1  /e *  term  in  (3.3).  For  the  evaluation  of  the  last 
term  in  (3.4),  first  use  the  law  of  the  mean  to  rewrite  it  as 

(3.5)  I  E«  Jdsft(Z£(u))E£  jdT  |Vx(X€(u-)  +  TdY£(u),  (£(s))  -  dY£(u)  . 

t<u<t+A  u  0 

Now,  by  a  change  of  scale  s/€2—s  and  the  use  of  (A3.1)  and  (A3.2)  again, 
we  see  that  this  term  is  0(e)  E£  [Y^  (t  +  A)  -  Yj(t)]. 

Putting  all  the  estimates  together  and  cancelling  the  ’!/«’  term  on  the  r.h.s. 
of  (3.3)  and  the  first  term  on  the  r.h.s.  of  (3.4),  we  get 

(3.6)  E£  f  €(t  +  A)  -  f€(t)  -  0(A)  +  0(c)  E*  (Y£(t  +  A)  -  Y£(t)) 

+  0(e)E£(Y£(t+A)  -  Y£(t)). 

Eqn.  (3.6)  yields  the  quasimartingale  and  the  uniformly  (in  e)  bounded  in  variation 
property  on  each  interval  [0,T]  for  f£().  By  letting  f(z)  «=  z  and  noting  that 
Z£(t)  =  0(0  ,  we  see  that  the  theorem  holds  for  the  (Z£(  ))  component.  Hence,  it 
also  holds  for  (X£(  )},  since  sup  E(Y^(t)  +  Y£(t))  <  *  for  each  t  and 

X£(t)  -  (  Z£(t)  +  Z£  (t))+  Y£(t)  -  Z£(t)  .  Q.  E.  D. 


We  summarize  (3.3)  to  (3.6)  for  future  use: 


nmmrtm 


*«t  wnoqOTWiOBW  nr.vmxA  iy.nwwmw  ka  wvkmw.w.  sr~  Kr^w\Fr^mMMmmmmm 
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t+A 

(3.7)  E£  f£(t  +  A)  -  f£(t)  -  |  E£  f,(  Z£(u))  G(  X£(u),  (£(u))du 

t 

t+A  T 

+  j;  Et  |  du  |  f„(  Z£(u))E£  F(X£(u),  (£(s))ds  F(X£(u),  <£(u)) 

t  u 

t+A  T 

+  ji  Ef  J  duE«  |  ft(Z£(u))Fx(X€(u),  ?€(s))ds  F(X£(u),(€(u))  +  0(A) 

t  u 

+  0(«)E£(Y£(t+A)  -  Y£(t))  +  0(«)Et£(Y£(t+A)  -  Y£(t)). 

Theorem  3.i  implies  that  (X£(  )  -  Z£,  Y^(  ),  Y£(  ),  Z£(  )  +  Z£(  )}  are 
quasimartingales  with  uniformly  bounded  variation  on  each  interval  [0,T]  and 
are  tight  on  D4[0,*]  in  the  pseudo-path  topology.  Hence,  the  same  tightness 
in  the  pseudo-path  topology  on  D4[0,«]  holds  for  {X  £(  )}.  [To  be  consistent 
with  the  usage  in  [11]  and  in  Section  2,  we  should  set  (w.l.o.g.)  X£(t)  -  0  and 

Z£(t)  -  0  for  sufficiently  large  t  ,  but  this  is  just  a  technicality  which  is 
convenient  for  the  statements  in  [11]  and  does  not  aTfcct  the  results.]  In  the 
next  theorem,  we  choose  and  work  with  a  weakly  convergent  subsequence,  also 
indexed  by  <  and  with  the  limit  denoted  by  X(  )  -  (X(  ),  Y0(  ),  Y,(  ),  Z(  )). 
Clearly,  the  sample  functions  Y.(-)  can  be  taken  to  be  non-decreasing  elements 
of  D[0,“).  Although  Y£(0)  -  0,  the  limits  Yj(  )  might  not  have  value  zero  at 
t  ■  0  .  To  account  for  this  possible  jump  in  the  integrals,  we  use  the 
normalization  Yj(0*)  ■  0  in  defining  integrals  with  respect  to  the  Y.(  ). 
(Recall  that  we  are  using  dY;(s)  -  Yj(s)  -  Yj(s'),  since  the  Y,(-)  are  taken  to 
be  right  continuous.) 

Since  the  pseudo-path  topology  is  equivalent  to  convergence  in  measure,  for 
almost  all  u,  t. 
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X(t)  -  Z(t)  +  Y0(t)  -  Y^t)  . 


In  fact,  (3.8)  holds  also  at  all  t  at  which  the  functions  arc  continuous.  In  the 
next  theorem,  we  obtain  the  stronger  and  more  useful  result  that  there  is  a  Wiener 
process  w(  )  such  that  X(  )  is  non-anticipative  with  respect  to  w(  )  and  (X(  ), 
Y0(  ),  Yx(-))  satisfies  (1.1)  for  that  w().  The  limits  Yj(-)  would  not  be  too  useful 
were  this  not  the  case.  We  use  the  following  ’ergodic’  type  assumptions. 


A3.3.  There  is  a  continuous  function  G(-)  such  that  for  each  x  , 


£  |  EuG(x  ,  Us))  ds  —  G(x) 


as  u  and  N  go  to  ^  . 


A3.4.  For  g  equal  to  either  F  or  F  and  T  >  u  +  N, 


E  Sup 

X 


J  EUS(*  • 


Us))ds 


as  u,  N,  T  go  to  “  . 


A3.5.  There  is  a  continuous  function  o(  )  such  that  for  each  x 


u+Tj 

f  J  Eu  F(x,  Ur))  dr  |  F(x,  Us))  ds 


- *  °2(x)/ 2 

as  T,  u  and  go  to  •  .  Also,  there  is  a  continuous  G0(  - )  such  that  for  each  x 


u+Tj  T+T 

f  EF(X  ,  UT))dT  [  F  (  x  ,  Us))ds-— *  G0  (x)  . 
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Remark.  If  $(•)  is  stationary,  then 

o2(x)  «  f  EF(x,?(0))  F(x,{(s))ds, 

G^x)  .  [  EF(x,?(0))  Fx(x,?(s))ds. 

Jo 

The  requirements  in  (A3.4),  (A3.5)  are  simply  conditions  on  the  rate  of  convergence 
as  r  -  u  ■*  •  of  the  conditional  expectation  of  functions  g(T),  of  the  noise  data  after 
time  r,  given  the  data  up  to  time  u. 


Theorem  3.2.  Assume  (A3.1)  -  (A3. 5)  with  sup£Ex[Yq(1)  +  Y*(t)]  <  “  for  each 
t  <  Then  {Z€(  - )}  is  tight  in  the  Skorohod  topology  on  D[0,“).  Any  weak  limit 
process  is  continuous  w.p.l.  Let  [X€( -),Yg(  ),Yj(  ),Z6( •)}  be  a  weakly  convergent 
subsequence  in  D^O,®).,  with  the  pseudopath  topology  used  on  the  first  three 
components  and  the  Skorohod  topology  on  the  last.  Denote  the  limit  by  X(  )  = 
(X(  ),Y0(  ),YX( -),Z(  ■)).  There  is  a  standard  Wiener  process  w(.)  such  that  X()  is 
non-anticipative  with  respect  to  w(  )  and 


G  (X(s))ds  +  G0(X(s))ds 


o(X(s))dw(s) 


Also ,  for  all  t,  w.p.l. 


(3.10)  X(t)  =  Z(t)  +  Y0(t)  -  Yj(t)  . 

[We  need  not  require  that  X €(t)  €  [0,B]  in  this  theorem.] 


Proof.  For  purposes  of  the  proof,  the  x-support  of  all  functions  can  be  taken 
to  be  compact,  and  the  Y;€(  ■ )  uniformly  bounded  (i.e.,  are  stopped  on  first 
reaching  some  large  value  N0).  The  general  case  follows  from  this  by  taking 
appropriate  limits  on  the  bounds.  Furthermore,  we  can  now  assume  that 
Z€(  )  is  bounded,  and  that  the  z-support  of  all  functions  is  also  compact. 
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Part  (a).  Tightness  of  {Z€ ( • )}  in  the  Skorohod  Topology.  Let  f(-),  f£()  and 
f£(-)  be  defined  as  in  Theorem  3.1.  We  use  the  perturbed  test  function  method 

of  [8]  or  [14]  for  proving  tightness.  Since  f£(t)  =  O(e),  Theorem  3.4  of  [8]  or 

Lemma  1  of  [14]  applied  to  the  perturbed  test  function  f£()  yields  the  tightness 
of  {f (Z€ ( - )}  in  D[0,®)  (Skorohod  topology)  for  each  smooth  f(),  hence  (Z£()} 
is  tight  on  D[0,®)  in  the  Skorohod  topology. 

(b)  The  limit  of  { Z£( •)}.  Fix  and  work  with  a  weakly  convergent 
subsequence  of  (X£(),  Yq(-).  Y£(-).Z£(-)}  also  indexed  by  e.  The  first  three 
components  converge  in  the  pseudopath  topology  and  the  last  in  the  Skorohod 
topology.  Let  h(-)  be  a  bounded  and  continuous  function.  For  0  <  k  <  “  and 
Aj  >  0,  tj  >  0,  let  us  define  H(  )  by 

H(  €,A  t  j  «  k)  =  Mfc.  Y£(s)ds,  i  =  0,1,  i  fJ  X£(s)ds,  i  P  Z£(s)ds,  j  <  k). 

j  J  tj-Aj  j  J  tj-Aj  j  J  t-Aj 

Let  tj  (  t  <  t  +  s,  for  all  j.  We  have  (See  (3.7)) 

ft+. 

(3.11)  lim EH(«,Aj,tj,  j  «  k)[f£(t+s)  -  f£(t)  -  Et£  (T£(u)  +  T^u)  +  T^u))du]  =  0, 

«  J  t 


where 


(3.12) 


Tj(u)  =  fl(Z£(u))G(X£(u),  (£(u)) 


fT 

T2(u)  =  €S  I  f,(Z€(u))  E£Fx(X£(u),  (£(s))ds  •  F(X£(u),  (£(u)) 

J  U 

rT 

T|( u)  =  x2  I  f„(Z£(u))  E£F(X£(u),  ?£(s))ds  F(X£(u),  (£(u)) 

*  U 

Fix  s.  Let  6£  -*  0  such  that  €*/6€  •*  0  as  €  -*  0.  Write  s  =  m£6£  and  suppose 
(w.l.o.g.)  that  the  m£  arc  integers.  For  the  purpose  of  evaluating  the  limit  in  (3.11), 
we  can  drop  the  f£(  )  components  of  the  f£(  )  (since  they  are  0(e)).  Also,  for 
the  same  purpose,  we  can  replace  the  integrals  J|+,E£T£(u)du  by 


(3.13) 


,  "V1  1 
if  Y.  -  •  6£ 

‘  j=o  6e  € 


.t+j8£+S£ 


E,‘+i6,  T>)du 


W  V  V  V-  V  vv  Vv/V  v\.  ■  .  'j-  '  .  ‘  >  V  Wv',ji'v ji"1  .v 


We  work  only  with  Tj(-).  since  the  others  are  treated  in  essentially  the  same 
way.  Rewrite  (3.13)  (with  i  =  3)  as 


t+j  6£+8£  T 

j  Et€+j6{  f  JZ£(u))dujF(X£(u),(£(s))dsF(X£(u),(£(u)). 

t+j&e  €  ^ 


Now,  change  scale  u/£i  -  u,  S/£2  -  s  and  define  s£  =  (t+j8£)/«2  and  let  (for 

A  ,  - 

notational  convenience)  E*  denote  the  expectation  given  (Y*(s),  s  <  t  +  j6€,  i  « 
0,1,  t(s),  s  <  s£).  Then  (3.14)  can  be  rewritten  as 


(3.15)  E£  "*£  8£  •  fJ+1E£fM  (Z£(e2u))  F(X£(£2u),  ?(u))du 

j=o  "£  i  e 

•j  ,T/€* 

F(X€(c 2u),((s))ds  - 


Due  to  (A3.4),  the  upper  limit  T/£2  can  be  replaced  by  any  large  Tj  and 
limits  on  Tj  taken  after  limits  on  e  are  taken. 

Fix  8-small.  Let  (Bi)  be  disjoint  intervals  covering  the  range  of  X£()  and 
with  diameter  less  than  and  let  x;  denote  an  arbitrary  point  in  Due  to  the 
uperossing  result  for  (X£()  +  Z£()}  implied  by  (2.2),  and  the  fact  that 
Z£(-)  *  O(e),  the  fraction  of  the  number  of  intervals  in  the  set  of  intervals 
{[t+j8€»  t+j6£+8£),  j  <  m€)  for  which  su|> |X£(t+j6£+u)  -  X£(t+j6£)|  >  8/2 
holds  goes  to  zero  in  probability  as  e  ■*  0.  Using  this,  the  tightness  in 
D[0,®)  (Skorohod  topology)  of  (Z£()}  and  (A3.4)  yields  that  the  limit  of  (3.16)  as 
e  -•  0  is  the  same  as  the  limit  of  (3.15)  as  e  -*  0. 

«  "If'1  t2  f'f+i 

<*•'«>  E«  8,  E  I(x«{l+j6()eBi)  *-  du 

A 

f„(Z£(t+j8£))E£F(Xi,((u))  F(Xj,((s))ds 


Now,  (A3. 5)  implies  that  the  limits  are  the  same  with  (3.17)  used  in  lieu 


of  (3.16). 
(3.17) 


Ee 


j=0  €  i  {X€(t+j6€)€Bi}  2 


+  j6t)) 


Define  the  operator  A(x)  by 

(3.18)  A(x)f(z)  =  f,(z)G(x)  +  f,(z)  G0(x)  +  fK(z)o2(x)/2  . 

Finally,  using  the  upcrossing  result  again  to  approximate  the  sum  of  the  (indicator 
functions  times  6e)  by  an  integral,  and  putting  the  above  estimate  together,  and 
using  a  similar  method  for  T*  and  T|  yields 

(3.19)  lim  EH(€,Aj,tj,  j  $  k)[f(Z€(t+s))  -  f(Ze(t))  - 

-  |  A(Xe(u))f(Ze(u))du]  -  0. 

Since  Z€(-)  converges  in  the  Skorohod  topology  on  D[0,*)  (and  we  have  not 
yet  proved  the  continuity  of  the  limit  Z(-))  the  set  I  of  t-points  for  which  P{Z(t)  * 
Z(t*)}  >  0  is  countable  but  need  not  be  empty.  Let  t  and  t+s  not  be  in  I.  The  triple 
(X€(-),  Yq(-),  Yj(-)}  converges  in  the  pseudopath  topology  and  the  integrals  in  H(  ) 
and  in  the  brackets  in  (3.19)  represent  functions  which  are  continuous  with  respect 
to  convergence  in  measure.  Then,  taking  limits  in  (3.19),  we  have 

(3.20)  EH(A.,tj,j  <  k)  [f(Z(t+s))  -  f(Z(t))  -  [t+A(X(u))f(Z(u))du]  =  0 

where  the  function  H(Aj,tj,j  <  k)  is  defined  to  be  just  H(e,A.,tj,j  <  k)  with  all 
functions  replaced  by  their  limits  as  t  -*  0. 

Owing  to  the  arbitrariness  of  k.tj.Aj  and  h(  )  and  of  the  points  s  and  t+s 
(not  in  I),  (3.20)  implies  that  for  each  smooth  f(  ),  the  process  defined  by 


f(Z(t))  -  [  A(X(u))f(Z(u))du  =  Mj(t) 

Jo 

is  a  martingale  with  respect  to  the  sequence  of  o-algebras  generated  by 
{X(s),Y0(s),Y1(s),  Z(s),  s  <  t).  The  fact  that  the  operator  A(x)  is  'local’  implies 
the  continuity  of  Z().  (See  a  proof  of  a  related  continuity  result  in  [8],  [14].) 

If  f(z)  =  z,  the  quadratic  variation  of  Mj()  is  J‘o2(X(u))du.  Owing  to 
these  facts  we  can  construct  a  standard  Wiener  process  w()  such  that 
X(  ),Z( -),Y0(  •)  and  Yj(  )  are  non-anticipative  with  respect  to  w(  )  and 

(3.21)  Z(t)  =  J  [5(X(u))  +  G0  (X(u))jdu  +  |  o(X(u))dw(u)  . 

It  follows  from  the  continuity  of  Z(  )  and  the  non-decreasing  property  of  the  Y(-) 
that  we  can  define  the  limit  X(  )  of  (X€(  )}  by  (3.10). 
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4.  The  Discrete  Parameter  Problem. 

The  discrete  parameter  analog  of  Theorems  3.1  and  3.2  is  obtained  very 
similarly  to  the  schemes  used  in  those  theorems,  and  we  discuss  only  a  few  of  the 
details,  for  one  discrete  parameter  form.  Just  as  for  the  continuous  parameter 
case,  the  general  ideas  are  applicable  to  a  much  broader  class  of  processes  than  used 
here.  Define  (X£)  by  Xq  =  x  and 

(4.1)  X«+1  =  X£  +  cG(X«,0  +  vn-F(X^)  +  6Y£  , 

where  we  define  6Y£  =  &Y5  -  6Y5,  and  &Y;€_  J  0.  Let  E£  denote  the 

n  On  In’  in  n 

expectation  conditioned  on  {X?,  j  <  n,  SY^,  SY£j,  £*,  j  <  n}.  Define  the 

n-l 

processes  Y£()  by  Y£(t)  =  E  BY^,  i  =  0,1,  and  X€(t)  =  X£  for  t  €  [nt,n€+€). 

j— o 

We  will  use 

A4.1.  sup  E  (Yq(1)  +  Y£(t))  <  •  for  each  t.  G(-,  ),  F(-,  )  and  Fx(-,-) 
are  bounded  and  measurable  and  the  latter  two  functions  are  continuous  in  x, 
uniformly  in 


A4.2.  For  each  x,  EF(x,(£)  =  0.  There  is  a  K  <  ®  such  that  for  all  N 
and  n  «  N, 


where  g  equals  either  F  or  Fx. 


Theorem  4.1.  Under  (A4.1)  and  (A4.2),  {X€(  ),Y£( •),  i  =  0,1)  is  ( with  the  possible 
addition  of  a  process  X£()  =  O(vT)  to  X£())  a  quasimartingale  with  variation 
uniformly  bounded  in  6  on  each  interval  [0,T], 


r 


utmiiiJimuiiuuiinwHWWW  r  Kvrjnenam  vxrmzr  rj  kw  uwF.mnuTvnmfTBmfwwBwn 
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Proof.  We  proceed  as  in  Theorem  3.1,  and  let  f()  be  a  function  that  is 
continuous  and  has  bounded  and  continuous  derivatives  up  to  order  three.  Fix  T, 
large.  For  N  =  T/€,  define 

f in  =  v'T  Z  E<  f.(Z*)F(X«,<f), 


j=n 


n  n'  x 


where  we  define  Z*  by  Z\  =  0  and  Z*+1  =  Z*  +  cG(X^,^)  +  /r  F(X*,S*). 


Define  the  processes  f*(-)  and  Z\{)  by  f*(t) 


f,€_  and  Z€(t)  -  Zt  on  the 


1  n 


n 

'€/ 


interval  [ne,n«  +  «).  Define  (as  in  Theorem  3.1)  f€(t)  =  f(Z€(t))  +  f*(t).  We 

show  that  f€(-)  is  a  quasimartingale  with  the  appropriately  bounded  variation. 
As  in  Theorem  3.1,  we  can  suppose  that  X€(  ),  Z€(-)  and  Yj€(-)  are  bounded  on 
[0,T]. 

With  a  rearrangement  of  terms,  we  can  write 


(4.3) 


E*fe(ne+0  -  f €(ne)  = 

E«[f(Z«  +  €G(X«,^)  +  vr  F(X^))  -  f(Z*)J 
-  vr  f,  (Z^)E'F(X^,^) 

+  ^  E  Entf.(Zn+X>  -  f,(Zn))  ’  F(X«+I,«J) 

j=n+l 

♦  "  f,<ZJ>  E  E'  |F(X;+1.lf)  -  F(Xj,{«)] 

j=n+l 


Via  a  truncated  Taylor  expansion,  we  see  that  the  sum  of  the  first  two  terms 
on  the  r.h.s.  of  (4.3)  equal 


(4.4) 


«  f,  <Z«>  E*  G(X«.tf  +  €  f„(Z')  E*  F2(X*,(*)/2  ♦  0(0- 


r*  c!rv£  »€\ 


Via  a  truncated  Taylor  expansion  and  (A4.2)  the  third  term  on  the  r.h.s.  of  (4.3) 
equals 


» 


-23- 


1 


(4.5)  €  ftt(Z«)  2  E£  F(X£+1,(£)F(X£,(£)  +  °(£)=  0(C) 

j=n+l 

Similarly  to  what  was  done  in  Theorem  3.1  to  the  ‘corresponding’  integral,  the  last 
term  in  (4.3)  can  be  written  as 


(4.6) 


**  f.(z 


n)K  2  f  E»+lFx(Xn  +  **2+1  -  X£),  $ j  )dT  •  (X£+1-X£) 
j=n+l  J0 


=  0(yr)E£(0(/r)  +  |6Y£|) 


Putting  all  the  estimates  together  yields. 


E£  f€(n€+€)  -  f€(n€)  =  0(c)  +  0(ye)E*  |6Y*  |. 


Letting  f(z)  =  z  yields  the  desired  result,  since  f£(  )  =  ()(✓?)  and 
sup  E(Yq(T)  +  Yj(T))  <  - 


Q.E.D. 


Theorem  3.2  can  also  be  carried  over  to  the  discrete  parameter  case.  We 
will  use  the  conditions. 

A4.3.  G(-,0  is  continuous  in  x,  uniformly  in  (.  There  is  a  continuous 

G( • )  such  that  for  each  x 

n+N  p  _ 

&  E  E£  G(x,lf)  - ’G(x) 

n 

as  n  and  N  go  to  •*>  . 
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A4.4.  There  are  continuous  R(j,x)  and  R0(j,x)  such  that  for  each  x 


m+N  p 

&  I  F(x,{<t,)F(x,^)  —  R(j,x) 


m+N  p 


05  m,  N  and  n-m  go  to  *  . 


A4.5.  For  g  equal  to  either  F  or  F  , 


E  sup)  E  E;g(x,5p| 

x  n+N, 


as  N,n  and  Nj  go  to  “  ( with  N  >  n+Nx)  . 


Define 


o2(x)  *  R(0,x)  +  2  E  R(j,x)  *  E  R(j,x) 


G  0(x)  =  E  R0(j,x) 


It  can  be  shown  that  (A4.5)  implies  that  the  sums  E  R(j,x),  E  R0(j,x)  converge 


uniformly  in  x  as  n  -*  •  (not  necessarily  absolutely). 


A  proof  parallel  to  that  of  Theorem  3.2  yields 


Theorem  4.2.  Assume  (A4.1)  to  (A4.5).  Then  the  conclusions  of  Theorem  3.2  hold 


for  the  model  (4.1). 


i.»Vf 
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5.  Auxiliary  Results 

In  this  section,  we  obtain  some  estimates  which  will  be  useful  in  Section  6, 

for  the  proofs  of  the  convergence  of  the  costs  Vq(x,Yq,Y£)  to  either  cost 

V0(x,Y0,Yj)  or  V(x).  We  will  show,  for  several  reasonable  classes  of  control 

policies,  that  sup  E|Y1e(t)|k  <  *  for  each  k  >  0  and  t  <  •  .  This  implies  the  uniform 
€  1 

integrability  property  needed  in  the  next  section:  In  Section  6,  we  will  need  to 

know  that  the  sequence  of  optimal  or  8-optimal  controls  for  x(  )  are  uniformly 

intcgrablc.  similarly,  we  will  need  to  know  whether  the  sequence  of  optimal  or 

8-optimal  controls  for  x€(  )  is  uniformly  integrablc. 

The  symbol  t£  will  denote  a  stopping  time  with  respect  to  either  of  the 

’data’  o-algebras  B{i;€(s),  s  <  t}  s  B*  or  B{(£,  «n  <  t)  £  B*  depending  on  the 

case,  and  we  write  El  and  Pi  for  the  expectation  and  probability,  conditioned 

(  1  € 

on  the  data  up  to  time  T£. 

Theorem  5.1.  Assume  either  (A3.1),  (A3.2)  or  (A4.1),  (A4.2).  Let  Q£(  )  and  Qfn  be 
bounded  and  B*  measurable  ( for  n«  <  t,  in  the  latter  case).  Define  X€(  )  and  X*  by 

(5.1a)  dX€  =  [G(X€,t€)  +  F(X€,(€)/€  +  Q€/«]dt 

(5.1b)  X<+1  -  X«  ♦  €G(X«,l||)  +  "  F(X‘,t‘)  +  VT  Q£n  • 

Define  Z€(  )  as  in  Section  3  or  4  ( continuous  and  discrete  parameters  case,  resp.).  For 
integer  k  and  t  <  ®  ,  there  are  ®  >  Kkt  -•  0  as  t  -•  0  such  that  ( for  small  (  >  0) 

(5.2)  E  sup  | Z€(r £  +  s)  -  Z£(T£)|2k  <  Km 

for  all  finite  (w.p.l)  T£. 


_ 
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Proof.  For  arbitrary  T  <  ■  define 
fT 

fjk€(t)  -  jt  2kZ«(t)2k-1Ef  F(X£(t),5£(s))ds/e  -  0(« ) | Z€(t) j 2k  l. 

The  right  hand  equality  is  a  consequence  of  (A3.2)  and  the  change  of  variable 
s/«2  -•  s.  We  do  the  proof  only  for  the  continuous  parameter  case  and  for  t£  *  G  - 
Z€(0)  *  0  for  simplicity.  The  proof  of  the  other  cases  is  essentially  the  same. 
Analogous  to  what  was  done  to  get  (3.4)  and  (3.7),  we  have 

ft+* 

(5.3)  Et€  [z€(t+s)2k  +  f2k£(t+s)]  -  [z£(t)2k  +  f2k£(t)]  -  E£  c2k£(u)du, 

•  Jt 

where 


T 

c2ok£(u)  =  M  2k[(2k-l)Z£(u)2k'2E£F(X£(u),(£(s))ds]F(x£(u),t£(u)) 

£  '  u 

T 

+  2kZ£(u)2k-1E£Fx(x£(u),(£(s))ds[Q£(u)  +  F(X£(u),t£(u))] 

fT/€2 

=  0(1)  Z€(u)2k*2E£F(X£(u),i(s))ds 

*u/€  2 

fT/€  2 

+  0(1)  Z£(u)2k-1E£Fx(x£(u),Us))ds. 

Ju/€2 

By  (A3. 2),  we  can  write  this  expression  as 


(5.4) 


Z£(u)2k'2  C2k€(u)  +  Z£(u)2k*1  C2k£(u) 

=  0(1)  ( I  Z£(u)  i 2k'2  +  I  Z£(u)  1 2k"!), 


where  the  Ck£(  )  arc  defined  in  the  obvious  way  and  arc  bounded. 

By  (5.3),  (5.4)  and  the  bound  f2k£(  •),  there  arc  •  >  K2k(T)  -  0  as  T  -*  0  and 
constants  K2'k  such  that,  for  t  <  T, 

E  [z€(t)2k  +  f2k£(t)]  <  K 2k  Jo  E  [l  +  |z£(s)|2k-1]ds  <  K2k(T). 

Define 

M2k€(t)  =  [z£(t)2k  +  f2k£(t)]  -  [  [z£(s)2k'2C2k£(s)  +  Z£(s)2k'1C2k£(s)]ds. 

■'o 


-27- 


By  (5.3),  M2k£(-)  is  a  martingale.  From  the  above  estimates  there  are  functions 
K.!Jk(t)  -*  0  as  t  -*  0  such  that  (use  Doobs  inequality  [15,  Theorem  7.3.4] 

(5.5)  E  sup  |  M2k€  (s)  1 2  <  4E|M2k£(t)|2  <  K"(t). 

»<t 

By  the  bound  on  f2k£()  and  (5.5),  we  get  (5.2)  for  r£  *  0.  Q.E.D. 

Theorem  5.2.  Assume  the  conditions  of  Theorem  5.1,  except  with  Q€(t)  -  0  (or  Q£n  - 

0)  for  t  >  T£.  Given  AQ  >  0,  there  are  60  >  0  and  T0  >  0  such  that  for  all  small  t 

(5.6)  P£  {sup  |z£(T£+t)  -  Z€(T£)  |  >  Aq}  <  1  -  60  . 

€  t*T0 

Proof.  The  result  follows  from  Theorem  5.1. 

Recall  the  definition  of  B  in  Section  1.  We  now  describe  some  classes  of 
controls  and  obtain  some  estimates  of  path  excursions  under  the  controls.  Let  L 
and  U  be  numbers  such  that  0  «  L  <  U  <  B.  Define 

dYg(t)  -  [F(Xf (t),l£(t))/ «  +  G(X£(t),(£(t))l  dt  I  £ 

L  J  (X£(t)=L) 

(5.7) 

dY£(t)  =  [F(X£(t),(£(t))/€  +  G(X£(t),(£(t))l  dt  I  £ 

1  L  J  (Xe(t)=U) 

For  obvious  reasons,  we  call  this  the  (L,U)  barrier  control  (following  the  usage  in 
[4]).  Define  the  discrete  parameter  barrier  policy  in  the  analogous  way:  the  dY£(  ) 
and  8Y£n  arc  just  large  enough  to  keep  X£(  )  and  X£  in  the  set  [L,U].  The  dY£/dt 
will  be  one  of  the  candidates  for  the  Q,  in  Theorem  5.1. 

Let  A0  <  B/2.  We  define  a  specific  control  policy  -  called  the  (B,A0)-control 
(for  the  continuous  parameter  case)  as  follows.  If  X£(t‘)  -  B,  immediately  set  Yj(t) 
■  Y£(t')  +  Aq  and  X£(t)  «  B  -  bQ.  Also,  Yq(  )  increases  just  fast  enough  to  keep 
X € (t)  >  0;  i.c.,  Y£(  )  is  given  by  (5.7)  for  L  »  0.  There  arc  analogous  definitions 
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and  results  for  the  discrete  parameter  process.  The  (B,A0)-control  has  some  nice 
properties  which  render  it  useful  for  the  discussion  in  the  next  section. 


Theorem  5.3.  Assume  the  (AQ,B )-control  and  either  (A3.1),  (A3. 2)  or  (A4.1), 
(A4.2).  For  each  t  and  integer  k. 


sup  E  |  Y*  (T£+t)  -  Yitrt)|k  < 

€,T£ 


Remark.  Owing  to  the  conditioning  in  (5.6),  the  estimates  for  Y£  are  proved 
almost  as  if  the  ’return’  process  from  the  point  (B-Aq)  to  (either  B  or 
B-2A0),  then  back  to  B-A0,  etc.,  were  constructed  from  a  Bernoulli  sequence. 


Proof.  We  do  the  continuous  parameter  case  only,  and  i  *  1.  The  case  i  =  0 
is  treated  by  an  argument  based  on  Theorems  5.2  and  5.4.  W.l.o.g.,  set  t£  =  0. 

Define  the  stopping  times:  of  =  min{t  >  0  :  X£(t)  =  B  -  A0)  and  for 

i  >  0,  pf  -  min(t  >  ofml  :  |X€(t)  -  (B  -  A0)|  >  A0},  of  =  min{t  >  pf  :  Xe(t)  = 

B  -  A0}.  We  will  estimate  the  kth  moment  of  N£(t)  *  max{i  :  of  (  t).  Define 

the  (0,1)  valued  random  variable  U£  as  follows.  Use  the  T0  of  Theorem 

5.2.  If  pf  -  of_t  <  T0,  set  U£  =  0  and  call  the  event  a  ’failure’.  If  pfm°f. i 
*  T0,  set  U£  =  1  and  call  the  event  a  ‘success’.  Let  N£  denote  the  number 
of  successive  passages  of  X£(  )  from  B-A0  to  either  B  or  B  -  2A0  which 
arc  failures,  after  the  ith  success.  Then 

t  ‘/To'1 

N€(t)  <  —  +  I  N£ 

To  o 

There  are  Kk  <  *»  such  that 

k  t/T  -l 

N£(t)k  <  +  Kk(VT0)  Z°  (Nf)k. 
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We  will  bound  E(Nj<)k.  Let  of  denote  the  return  time  of  X£()  to 
B  -  A0  immediately  after  the  ith  success.  Then,  by  Theorem  5.2, 

P*€  (N£ >  n}  - 

^0€  ^0tj+j+i  ‘  0»j+j  <  To.  j  <  n} 

<  (1  -  V"'1 

This  yields  E(Nf)k  <  (constant)/60,  and  the  proof  is  concluded,  since  Y£(t)  < 
A0N£(t). 

Q.E.D 

Theorem  5.4.  Assume  either  (A3.1),  (A3.2)  or  (A4.1),  (A4.2)  and  the 

(L,U )-barrier  policy.  Then  for  each  t 

(5.9)  sup  [E(Y£(t+T)  -  Y£(T))2  +  E(Y£(t+T)  -  Y£(T))21  <  -  . 

€  ,T  J 

Proof.  Again,  we  do  only  some  of  the  details  for  Y£(  -),  and  for  the 
continuous  parameter  case.  Drop  the  G(-,-)  for  notational  simplicity.  Denote 
the  initial  time  by  tQ  and  let  A0  <  (U-L)/2  and  define  the  stopping  times 

o£  =  min{t  >  t0  :  X£(t)  -  L} 
and  for  i  >  0, 

o£  =  min{t  >  pf  :  X£(t)  -  L},  pf  -  min{t  >  of.,  :  X£(t)  *  L  +  A0}  . 

Set  the  stopping  times  to  06  if  they  arc  not  otherwise  defined.  All  the  needed 
estimates  can  be  shown  to  be  uniform  in  t0  and  we  set  t0  *  0  for 
simplicity. 

We  can  write  (and  simultaneously  define  Z£()) 


•/ *Jr  0  C*  •  *  *  •  •  •  *  0  *  «  ,  *  .  •  .  *' m  «Ti  r  «  m  < m  m'  .  -• 


•  .4  ' 
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(5.10) 


I  [x£(p£+l  n  t)  -  X£(o£n  t)]  - 

i 

E  [z£(p£+1  n  t)  -  Z £(o£  n  t)]  +  Y£(t) 

i 

-  Z£(t)  +  Yg(t)  . 


The  mean  square  value  of  the  term  on  the  left  of  (5.10)  is  bounded  above  by 
Aq  times  the  expectation  of  the  square  of  the  number  of  i  for  which  p5  i  t.  By 
an  argument  very  similar  to  that  used  in  Theorem  5.3,  this  can  be  shown  to  be 
bounded  uniformly  in  e  for  each  t. 

Define  M£(t)  *  Z£(t)  -  JgC£(s)ds,  where  Z£()  is  defined  in  Theorem  3.1 
or,  equivalently,  it  is  the  fj£()  of  Theorem  5.1.  The  C£()  is  defined  in 
Theorem  5.1.  The  C2£()  defined  there  doesn’t  appear  here,  since  2k  =  1  here. 
Define  N£()  as  in  Theorem  5.3.  Then,  since  M£(  )  is  a  martingale  on  the 
interval  where  dY£(  )  *  0  we  have, 

E(E[m£(P£+1  n  t)  -  M£(o£n  t)])2 

-  I  E |M€(p£+1  n  t)  -  M£(0£  n  t)  I2 

(5.11) 

-  0(1)  E(sup  |Z£(s)|2  +  1)  N£(t) 
s«t 

*  0(1  )E^  (sup  | Z£(s) ] 4  +  1)  E**  | N£(t)  | 2  <  K,  <  »  . 
sSt 

The  last  inequality  follows  from  Theorems  5.1  and  5.3.  Since  Cj€(  )  =  0(1),  and 
Z£(-)  »  O(t),  and  sup£E(N£(t))2  <  •  ,  there  arc  K2  <  "  K3  <  ",  such  that  the  left 
side  of  (5.11)  can  be  bounded  below  by 


a 


s 

1 


K2E  (E  [z£(p£+1  n  t)  -  Z£(o£  n  t)])2  -  Ks 


K2  E  j  Z£(t)  | 2  -  k3  . 


The  proof  that  sup  E  |  Z€ (t)  | 2  <  "  follows  from  these  inequalities. 


Q.E.D 
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6.  Convergence  of  the  Costs  and  Controls. 

In  [1],  it  is  shown  that  there  are  0  <  L*  <  U*  <  ®  such  that  (under 

appropriate  conditions)  the  optimal  control  for  (1.1)  is  a  (L*,U*)-barrier  control.  We 

assume  that  B  is  large  enough  so  that  U*  <  B.  Let  Y;(),  i  *  0,1,  denote  this  optimal 
control.  The  set  of  increments  of  the  ‘local  time’  control  processes  (Yj(n+1)  -  Y;(n), 
i  »  0,1,  n  <  “}  are  uniformly  integrable.  Let  Yj€(  ),  i  =  0,1,  denote  the 

(L*,U*)-barrier  control  for  X€()  (continuous  or  discrete  time).  The  following 
theorem  says  that  the  optimal  control  for  X(  )  is  ‘nearly’  optimal  for  Xe(  ). 

Theorem  6.1.  Assume  either  (A3.1)  to  (A3.5)  or  (A4.1)  to  (A4.5).  Let  (1.1)  have  a 
unique  weak  sense  solution  for  the  (L *,V*)-barrier  policy,  and  let  this  policy  be 
optimal.  Then  (X€( -),  Y«(  ■),  Y*(  •))  *  (X(  •),  YQ(  • ),  Yx(  ■ ))  in  the  pseudopalh 
topology,  and  there  is  a  Wiener  process  w(-)  such  that  (X(  ),  Y0(-),  Yj(  ))  is 

non-anticipalive  with  respect  to  w(),  and  (1.1)  holds.  Also 

(6.1)  V«(x,  Yg,  Y\)  -  V0(x,  Y0,  Yj)  =  V(x). 

For  6  >  0,  let  Yq(  ),  Y*()  be  B-oplimal  policies  for  X€(  )  such  that  (6.2)  is 
uniformly  integrable. 

(Yj€(n+1)  -  Y«(n),  €  >  0,  n  < 


(6.2) 

Then 

(6.3) 


6  +  lija.  v«(x)  )  'if-  V«  (x.Y^.Yf)  >  V(x)  . 


\ 


i 

S.’ 

S 

> 

> 

\ 

i 

'  * 


. . ***** 


Remark  on  (6.2).  The  uniform  integrability  is  used  basically  to  ass  ire  that  the  cost 
associated  with  the  limit  process  is  the  limit  of  the  costs  associated  with  X€(  ).  We 
have  not  been  able  to  prove  the  theorem  without  it,  unless  all  cost  terms  are 
positive  (see  Theorem  6.2).  With  the  cost  structure  used  here  and  in  [1],  it  is 
conceivable  (in  that  we  have  not  yet  proved  otherwise)  that  as  €  -»  0,  the  increments 
in  both  Yq(  )  and  Y*()  grow  without  bound.  But,  (as  shown  in  Section  5)  this 
won’t  happen  for  a  large  class  of  reasonable  controls.  The  uniform  integrability 
holds  for  a  wide  variety  of  control  processes:  E.g.,  for  (1)  Combinations  of  the 
(L.U)-barrier  and  (B,AQ)-policies  (Theorems  5.3  and  5.4);  (2)  these  theorems  can  be 
extended  to  cover  the  case  where  there  arc  numbers  L°,  U°,  AQ,  Aj  where  A0  +  A1  < 
(U°  -  L°)/2  and  Y*()  acts  only  in  [L°,L°  +  A0],  Y*()  only  in  [U°  -  A0,  U°],  and  with 
maximum  jump  «  Ax;  (3)  Let  Yf()  denote  any  admissible  policy  and  fix  N. 
Define 

T‘  =  min{t  >  n:  (Y«(t)  -  Y«(n))  +  (Y«(t)  -  Y«(n))  >  N)  n  (n  +  1). 

On  the  interval  [n,  n+1),  use  Yf{  )  on  [n,  T*),  then  switch  to  a  barrier  or  (B,AQ) 
policy  on  [t*,  n+1).  In  Theorem  6.2,  it  is  shown  that  (6.2)  is  not  needed  if  -kjdY^t) 
is  replaced  by  the  positive  cost  increment  k1dY1(t). 

Proof.  We  do  only  the  continuous  parameter  case.  Let  X£(  )  denote  the  process 
with  the  Y€(  )  used.  By  Theorem  5.4,  (Yj€(n+1)  -  Yje(n),  t  >  0,  n  <  °°)  is  uniformly 
integrable.  Extract  a  weakly  convergent  subsequence  of  (X€(),  Yq(), 

Y*()}  (pseudopath  topology)  and  denote  the  limit  by  (X(-),Y0(-),Y1(  ■))•  By 

Theorem  3.2,  this  triple  satisfies  (1.1)  for  some  w(  ).  Clearly,  the  Y^-^Y^-)  is 

«  *  *  ♦ 

the  (L  ,U  )-barricr  policy,  since  it  can  only  increase  when  X(t)  =  L  or  U  as 

appropriate.  Hence  Y;(  )  =  Yj(  ).  By  this  and  the  uniqueness  of  the  solution  to 
(1.1),  the  limit  docs  not  depend  on  the  chosen  subsequence. 


By  the  uniform  integrability  asserted  in  the  above  paragraph, 


(6.4)  lim  Vq(x,Yq,Yj)  =  lim  E  (“  e‘et  [k0dY«(t)  -  k^Y^t)  +  k(Xfu))dt] 

-  E  J“  e-6t  [k0dY0(t)  -  kjdY^t)  +  k(X(t))dt] 

=  V^x.Yq.Y,)  =  V(x)  . 


To  get  (6.3),  repeat  the  procedure  with  controls  Y|(-),  Y*().  Here,  the 
limit  (X(  *),Y0(  -)»YX(  •))  might  depend  on  the  chosen  subsequence.  But,  for  any 
convergent  subsequence  {«n}  we  get  lim  V*(x,Yq,Yx)  -*  V0(x,Y0,Y1)  >  V(x).  Hence, 

6  — €n"*0 

by  the  definition  of  6-optimality  and  the  weak  convergence. 


8  +  Uf-  Ve(x)  *  toL  Vq(x,Y q,Y *) 

>  inf  V0(x, Y 0,Y j)  =  V(x). 
Yo,Yi 


Q.E.D. 

Theorem  6.2.  Assume  the  conditions  of  Theorem  6.1,  except  for  the  uniform 
integrability  of  (6.2),  but  let  the  cost  be 

Ex  e-Bt[k0dY0(t)  +  kx  dY j(t)  +  k(x(t))dt]  =  V0(x,Y0,Yx), 

Jo 

and  similarly  define  Vq(x,Yq,Yx),  where  k;  >  0.  Then  the  conclusions  of  Theorem  6.1 
( with  the  8  in  (6.3)  replaced  by  28)  hold. 

Proof.  Let  Y.e(-),  i  «  0,1,  denote  a  6-optimal  policy.  We  can  suppose  that 
sup[ExYQ(t)  +  ExYj(t)]  <  •  for  each  t  <  •  and 

lim  sup  e-fit[k0dY^(t)  +  kxdY*(t)  +  k(X€(t))dt]  =  0, 

T  €  Jip 


-34 


since  this  holds  for  any  barrier  policy.  In  fact,  there  is  a  Ng  <  ®  such  that  if  we 
switch  to  the  (L*,U*)  barrier  policy  (or  to  any  barrier  policy)  once  the  Yj€(t) 
exceeds  N&,  we  change  the  cost  by  less  than  6.  But,  then  the  set  (6.2)  is  uniformly 
integrable,  and  Theorem  6.1  holds. 

Q.E.D. 
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7.  Average  Cost  Per  Unit  Time. 

The  methods  of  Sections  1  to  5  can  be  used  to  adjust  the  proof  of  Theorem  8 
in  [6]  to  get  the  result  which  is  analogous  to  Theorem  6.1  for  the  average  cost  per 
unit  time  problem.  Only  an  outline  of  the  method  will  be  given.  The  reader  is 
referred  to  the  reference  for  more  details  on  the  structure  of  the  approximation  for 
the  average  cost  problem  for  the  non-singular  case  (and  which  can  be  carried  over 
to  our  case). 

For  the  average  cost  per  unit  time  problem,  we  wish  to  work  with  feedback 
controls  and,  hence,  use  only  Yf(-),  i  =  0,1,  or  Y;(-),  i  =  0,1,  for  which  the 
associated  processes  5€(-)  and  (Xe(  •))  or  X(-),  resp.,  are  bounded 

Markov-Feller  processes.  Also,  let  U€(t),  e  >  0,  t  <  “}  be  bounded.  The  cost 
criteria  arc 


Hm  E  T  [T  [MY0(t)  -  MY,(t)  +  k(X(t))dt]  s  7(Y0,Y,) 

T  Jo 

Km  E  £  T  [k0dY0€(t)  -  kjdYjd)  +  k(X€(t))dt]  =  7€(Y0,YX)  . 

T  Jo 

For  simplicity,  we  do  only  the  continuous  parameter  case.  The  discrete 
parameter  case  uses  very  similar  assumptions  and  proof.  Let  PM  (PM€,resp.) 
denote  the  class  of  feedback  control  processes  for  which  X()  (resp., 
(X€( -),5€( •))  is  a  Markov-Feller  process.  Let  NA  (resp.,  NA€)  denote  the  class  of 
non-anticipative  controls.  We  will  use  the  following  assumptions. 

A7.1.  There  is  an  >  0  such  that  f or  each  8  >  0  and  e  (  «0,  there  are 
8- optimal  controls  €  PM€  of  the  form 
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(7.1)  dYf  =  Q f  (x,()dt,  i  =  0,1, 
where  the  Qf(-,  )  are  continuous. 

Note:  If  Q£(x,5)  is  Lipschitz  continuous  in  x,  uniformly  in  5,  then  Yf(-)  €  PM£. 
See  the  remark  below  where  it  is  shown  that  the  barrier  and  (B,A0)  policies  can  often  be 
smoothed  to  yield  a  continuous  Q£(-,). 

A7.2.  A  (L*,U*)  barrier  control  Yf)  is  optimal  for  (1.1),  and  (1.1)  then  has  a  unique 
invariant  measure.  This  control  is  in  PM  and  its  adaptation  Y£(  •),  i  =  0,1,  to  Y£()  is 
in  PM£.  When  applied  to  PM£,  (X£(  ),(£(  •))  has  a  unique  solution  and  invariant 
measure. 

A7.3.  inf  7(Y0,Yj)  =  inf  7(Y0,Y t)  . 

Yj€  PM  Yj£NA 

Theorem  7.1.  Assume  (A3.1)  to  (A3.5)  and  (A7.1)  to  (A7.3).  Let  G(x,()  be 
Lipschitz  continuous  in  x,  uniformly  in  {  For  6  >  0,  let  Y£(  ),  i  =  1,2,  be  a  sequence 
of  6- optimal  controls  in  PM£  (for  X£())  and  let  (the  Y£()  are  of  the  form  discussed 
in  (A7.1))  with  Q£  associated  with  Y£ 

(7.2)  (Y£(n+1)  -  Y£(n),  €  >  0,  n  <  »,  X€(0)  =  x,  (£(0)  =  (} 
be  uniformly  inlegrable.  Then 

6  +  Uf  -  7€(Y£,Y£)  >  lim  7€(Yo  >  Y£)  =  7(Y0,Y1). 

Remark.  The  (L*,U*)  barrier  control  can  be  approximated  for  X£(  )  in  such  a  way 
that  it  is  of  the  form  in  (A7.1).  In  particular,  let  A£  •*  0  as  e  -  0  and  define 
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dY  f(t)  -  dt[F(X«(t),5€(t))/€  +  G(X£(tUe(t))] 


[X£(t)  -  U*  +  A,] 


{X€  (t)£  (U*-Aj  ,U*)} 


and  similarly  for  Yg().  It  can  be  shown  that 


7€(Yo-Y^  >€<Yo.Yi€). 


where  the  Y£  is  the  (L  ,U  )-barrier  policy  for  X£().  Clearly,  the  Y£(  )  are 
of  the  form  used  in  (A7.1).  By  (7.4),  for  each  €,  we  can  choose  A€  so  that  the 
left  and  right  sides  of  (7.4)  are  as  close  as  desired.  By  using  techniques  of 
Section  5,  it  can  be  shown  that  (Y£(n+1)  -  Y£(n),  i  =  0,1,  n  <  *,  €  >  0, 
X£(0)  =  x,  ?£(0)  =  l)  is  uniformly  integrable. 


Proof.  For  each  €,  6,  T,  define  the  measure 

<»T 

p£'8()  =  £e  P€’8(X€(0U£(0),t,-)dt, 

•*0 

where  P£’8  is  the  transition  function  for  (X£(  ),t€( •)),  under  the  6-optimal 
control  Y£(  •)  (or  Qf)  of  (A7.1).  Then 

7€(Y£,Y£)  -  J  P£>8(dxdOIk0Q£(x,O  -  k^x.O  +  k(x)]. 

Choose  a  subsequence  T  -*  •  such  that  both  the  lim  is  attained  and 
P£’8  converges  weakly  (with  limit  denoted  by  f*£’8).  Then,  by  the 
Markov-Feller  property  of  (X£(),  {*(•))  for  (Y|(-),  Y£())  €  PM£,  ji€’6  is  an 
invariant  measure  for  (X€(  ),?€( •))  and  (by  the  continuity  of  the  Q£  and  the 
weak  convergence, 

(7.4)  7€(Yq,Y£)  =  [  M€,5(dxd()  [k0Qo(x,0  -  kjQftx.l)  +  k(x)]  . 


■wirw  if*  ywtruLr^  omw  inr  hti  iwum  w*  gw  vw  vnvwvnir^irwjirjr^r^.ir^r*.  v^ir^jrur^jr^jr»-jir^t™ 


KSttKttSfo: 
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a  ^  a  ^ 

Let  (X'(  •),{(•))  denote  the  stationary  process  associated  with  the  controls 

A 

Qi€(-»  )  a°d  measure  m€,6()  and  let  Y£()  denote  the  corresponding  stationary 
control  processes.  Then  we  can  write  (7.4)  as: 

7€(Y£Y£)  =  E  pdt^Q^X^tX^d))  -  kjQ^X^tX^d))  +  k(X  £(t))]. 

Jo 

(7-5)  fl  A 

=  E  k(X£(t))dt  +  Ek0Y£(l)  -  EkjY^l). 

A  A 

By  the  uniform  integrability  (7.2),  (Y£(l),  Y£(l),  «  >  0)  is  uniformly  integrable. 

A 

Now,  choose  a  weakly  convergent  subsequence  of  (X€(  ),Y£(  ),Y£(  •  )},  with 

A  A  A 

limit  denoted  by  (X(  ),Y0(  J.Y^ •)).  The  limit  is  stationary ,  satisfies  (1.1)  and 
(indexing  the  subsequence  by  e  also),  we  have 

7€(Y£,Y£)  -  tfYo.Yj)  >  7(Yq,  Y,), 

where  the  optimality  of  Y0(),  Yj(  )  is  used. 

The  proof  is  concluded  by  applying  the  same  procedure  to  Y£(  ),  Y£(  ), 
where  the  ‘smoothing  interval’  (see  remark  above  the  theorem  where  Y£  is 
defined)  goes  to  zero  fast  enough  as  e  -*  0. 

Q.E.D. 
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8.  The  Vector  Case.  Formulation  and  Quasimartingale  Estimates 
and  the  Approximation  Theorem 

Most  of  the  foregoing  analysis  and  results  can  be  carried  over  to  the 
case  of  vector  (x  €  Rr,  Euclidean  r-space)  valued  G,  F  in  (1.4)  or  (3.1).  Since 
the  details  of  the  proofs  are  essentially  the  same  as  in  the  foregoing  sections, 
only  an  outline  will  be  given.  Only  the  continuous  parameter  case  will  be 
discussed,  but  under  the  obvious  changes  in  the  assumptions  (A3.1  to  A3.4)  and 
(A8.1)  used  below,  the  discrete  parameter  results  also  extend  to  the  vector  case. 

Applications  to  queueing  and  production  networks  require  a  somewhat  more 

special  development,  and  this  will  be  published  spearately. 

We  use  the  model  (vector  F,  G) 

(8.1)  dX£  =  [G(X£,?£)  +  F(X€,?€)/€]dt  +  dY£(t), 
with  cost 

00 

(8.2)  V£(x,  Y£)  =  f  e-6l(k0  |  dY£(t)|  +  k(X£(t))dt],  0  >  0, 

,lo 

The  results  of  Section  7  can  also  be  extended  to  the  vector  case. 

Thorcm  8.1.  Assume  (A3.1,  A3. 2)  with  vector  G,  F  used,  and  let  sup€EjJ|dY£(t)  <  00 
for  each  T  <  *.  Then  ( with  the  addition  of  a  process  whose  maximum  value  goes  to 

zero  as  e  0)  (X£(  ),  Z£(  ),  Y£(  ))  are  quasimartingales  with  uniformly  ( in  e) 

bounded  variation  on  each  bounded  time  interval. 

Remark.  The  proof  is  essentially  identical  to  that  of  Theorem  3.1.  Similarly,  the 
proof  of  Theorem  8.2  below  is  essentially  identical  to  that  of  Theorem  3.2. 

We  will  next  use  (A8.1),  the  vector  form  of  (A3. 5). 


A8.1.  There  is  a  matrix  E(  )  with  a  continuous  and  bounded  square  root  o(  ) 


such  that  for  each  x, 

1  fu+Ti  fT+T 

—  EuF(x,5(r))dT  F'(x,?(s))ds 

1  X  Ju  * T 

1  ffu+Tl  fT+T  l'  P 

+  — [J  EuF(x,4(T))dTj  F'(x,Us))dsJ  — *  E(x), 

as  T,  u  and  T l  go  to  m  .  There  is  a  continuous  G0(  • )  with  components  G0i(  • ),  i  <  r, 
such  that  for  each  x, 

1  fu+Ti  fT+T  P  - 

—  ZE^x.UT^dT  Fix  (x,Us))ds — *  Ga(x) 

1  1  Ju  j  •’T  ■> 

as  T,  u  and  Tj  go  to  «  . 

Theorem  8.2.  Assume  (A3.1)  to  (A3.4),  and  (A8.1)  and  let  sup€ExJJ|dY€(s)|  <  •  for 
each  T  <  *  .  Then  {Z€(  - )}  is  tight  in  the  Skorohod  topology  on  Dr[0,°°)  and  any  weak 
limit  process  is  continuous  w.p.l.  Let  (X€(  ),Y€(  ),Ze(  •)}  be  a  weakly  convergent 
subsequence  in  D3r[0,“),  with  the  pseudopalh  topology  used  on  the  first  two  components 
and  the  Skorohod  topology  on  the  last.  Let  X(  )  «  (X(  -),Y(  ),Z(  •))  denote  the  limit  of 
a  weakly  convergent  subsequence.  Then  the  conclusions  of  Theorem  3.2  continue  to  hold, 
with  the  limit  Y(  )  replacing  Y0(-)  -  Yj(-).  In  particular,  the  limit  satisfies 

(8.3)  X(t)  *  Z(t)  +  Y(t)  +  X(0),  dZ(t)  =  [G(x)  +  G0(x)]dt  +  o(x)dw, 

Z(0)  -  0. 

Definition  and  Assumptions.  Below,  v(  )  will  be  a  continuous  vector  field  on  Rr 
with  |v(x)|  s  1,  and  S  a  compact  set  with  a  piecewise  differential  boundary  and 
with  the  following  property:  There  is  a  bQ  >  0  such  that  for  x  €  9S  and  A  «  A0,  the 
points  x  +  Av(x)  arc  interior  to  S.  Define  the  (S,  A,  v(  ))-reflccting  policy  for 


41 


X£()  as  the  (admissible)  policy  which  sets  X€(t)  =  x  +  Av(x),  if  X£(t')  =  x  €  9S. 
Then,  of  course,  dY£(t)  *  Av(x).  The  same  definition  is  used  for  the  (S,  A,  v(  ))- 
reflecting  policy  for  the  X(  )  of  (8.3). 

A  policy  Y(  )  for  (8.3)  is  called  a  (S,  v(  ))-reflecting  policy  if  the 
associated  process  X(  )  is  a  reflected  diffusion  in  S,  with  continuous  reflection 
direction  v(  )  on  9S,  and  there  is  a  A0  >  0  such  that  for  A  «  V  the  policy  which 
sets  X(t)  ■  x  +  Av(x)  if  X(t')  =  x  €  9S  is  an  admissible  (S,  A,  v(  ))-reflecting  policy. 

Theorems  5.1  and  5.2  continue  to  hold.  Here,  we  would  require  that  the 
Qe  of  these  theorems  be  such  that  it  guarantees  boundedness  of  the  X£(  );  e.g., 
choose  a  bounded  set  S,,  and  let  Qe  simply  just  push  ‘enough’  to  keep  X£(  )  from 
leaving  that  set.  In  the  approximation  Theorem  8.5,  we  use  a  different  approach, 
based  on  the  use  of  a  (S,A,v(  ))-reflecting  policy  to  approximate  a  ‘reflecting’ 
diffusion.  Wc  will  use  only  the  following  two  theorems. 


Theorem  8.3.  Assume  (A3.1)  and  (A3.2)  ( vector  case),  and  let  Y£,A(  )  denote  a 
(S,A,v( -))-rc fleeting  policy  for  X£().  Then  for  each  T  <  “  and  integer  k 


T  1 

■m.  E4LldY<H 


Theorem  8.4.  Assume  ine  model  (8.3)  with  bounded  and  continuous  G(  ),  G0(  )  and 
o(  ).  Let  Ya(  )  be  a  (S,A,v(  ))-;e fleeting  policy  for  x(  ).  Then,  for  each  T  <  *  and 
integer  k, 

T 

IS£0  M  J0 ldYi(s)lJ  '*• 

x€S 

Wc  will  prove  Theorem  8.4  only.  The  proof  of  Theorem  8.3  is  similar,  and 
uses  the  (vector  case)  estimate  (5.6)  for  the  process  Z£(t)  +  Z[(t),  where  Z£(  )  is  the 
appropriate  ‘vector’  case  replacement  for  the  Zj(-)  used  in  Sections  3  to  5. 
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Proof  of  Theorem  8.4.  Let  YA()  denote  the  (S,A,V(  ))-reflecting  policy  and  XA() 


the  associated  solution  to  (8.3).  Fix  e  >  0  and  small.  Let  Na(x)  denote  the 
a-neighborhood  of  x.  There  are  x2,  ....  xq  on  9S  such  that  U*=1  Na(Xj)  D  9S  and 


sup  ,  %  I  v(x)  -  v(y)  |  <  a 


sup 

t,y€NJa(xi 


Let  oJJ1  denote  the  first  time  of  entry  of  XA(-)  into  N^x^.  Define 


p["  «  min{t  >  oj"^:  XA(t)  £  N2a(xm)} 


o?  =  min(t  >  p["  :  XA(t)  €  Na(xm)}. 


Define  NA  =  max{i:  o'"  S  T),  Yf-m(T)  =  YA(p|"+1  n  T)  -  YA(c^  n  T)  and  YA>m(T)  = 

IjYf'^T). 


Owing  to  (8.5)  and  the  smallness  of  a,  there  is  a  Kt  <  «*  (depending  on  ot. 


but  not  on  A)  such  that 


rT  q 

I d Ya(s)  I  <  K.  I  |  YAm(T)  |. 
» n  m=l 


Hence  we  need  only  evaluate  E[YA,m(T)]\  We  have 


m 

YA,m  =  I  [XA(p["  n  T)  -  xA(o™  n  T)] 


-  E™  [ZA(p|"  n  T)  -  ZA(oPj  n  T)]. 


The  absolute  value  of  the  first  term  on  the  r.h.s.  of  (8.6)  is  <  ocK,NA  for  some 


constant  K2  <  •  .  The  absolute  value  of  the  last  term  on  the  r.h.s.  of  (8.6)  is 


bounded  above  by 


(8.7)  Na  •  supT  |zA(t)  -  ZA(s)|. 


Since  zA(  )  is  just  the  sum  of  an  ordinary  integral  and  a  stochastic  integral 


whose  integrands  arc  bounded  uniformly  in  A,  all  the  moments  of  the  last  factor  in 


LvM 


(8.7)  are  bounded  uniformly  in  A.  Hence  it  is  enough  to  show  that  sup  EJN^|P  <  • 

xes,A 

for  all  integers  p. 

This  last  problem  is  similar  to  that  dealt  with  in  Theorem  5.3.  Owing  to 
the  nature  of  the  (S,A,v(-))  -  reflecting  policy  there  is  an  a'  >  0  (not  depending  on 
A)  such  that  in  order  for  XA(-)  to  move  from  the  exterior  of  N20((xm)  at  time  p["  to 
Na(xm)  at  time  o?",  we  must  have  sup  |ZA(t)  -  ZA(p[")|  *  a'.  Let  T  be  a  finite 
stopping  time.  For  each  6Q  e  (0,1),  there  is  a  T0  >  0  such  that  for  all  small  A  >  0, 


sup  Px  {  sup  |zA(t+T)  -  ZA(T)I  >  a 72}  <  1  -  60 

k€S  ,T  t 


The  inequality  (8.8)  and  an  argument  like  that  used  in  Theorem  5.3  (to  get  the 


upper  estimate  on  EIN^ (k  there)  completes  our  proof. 


Q.E.D. 


Definitions.  We  now  add  an  additional  qualification  on  the  control  problem.  It  is 
supposed  that  there  is  a  compact  set  S1  with  a  piecewise  differentiable  boundary 
such  that  X€(  )  and  X(  )  arc  to  be  confined  to  Sr  Let  there  exist  a  (S1,A,v1( -))• 
reflecting  policy  for  small  A  and  some  continuous  Vj(  ).  As  noted  in  the  remarks 
after  the  theorem,  the  approximation  theorem  is  easier  to  prove  without  this 
restriction.  In  the  theorem,  we  assume  that  the  optimal  control  for  X(  )  is 
(S,v( •  ))-rcflccting  for  some  S  (compact,  since  Sj  is  compact).  This  will  often  be  the 
case.  But,  as  noted  in  the  introduction,  other  forms  are  possible:  combined  singular 
and  non-singular  controls,  true  impulsive  controls,  etc. 


Theorem  8.5.  Assume  (A3.1)  -  (A3. 4)  and  (A8.1),  and  the  condition  in  the  above 
paragraph.  Suppose  that  the  optimal  policy  Y()  for  X(  )  is  a  (S,v(  ))-re fleeting  policy 
for  some  bounded  S.  Let  YA(0  denote  its  (S,A,v(  ))  reflecting  form.  Let  (8.3)  have  a 

^  /  A 

unique  weak  sense  solution  under  both  policies,  for  all  small  A.  Let  Y*>  (•)  denote  the 
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(S.A, v(  ))-r effecting  policy  adapted  to  X£().  Given  6  >  0,  there  is  a  tv  >  0  such  that 
Y£,A()  is  2S-optimal  for  X£()  and  small  t  in  the  sense  that 

(8.9)  6  +  lim  V£(x)  >  limV‘(x,  Y£-A)  «  V0(x,  Y  A),  V0(x,YA)  <  V(x)  +  6. 

Proof.  The  method  is  that  of  Theorems  6.1  and  6.2.  Let  Y£()  denote  the  optimal 
(or  8/4-optimal,  if  there  is  no  optimal  policy)  policy  for  X£()  and  X£,A()  the 
process  corresponding  to  Y  £  ,A(  - ).  By  the  argument  of  Theorem  6.2,  there  is  no  loss 
of  generality  if  we  suppose  that  the  second  set  of 

[[  +1  |dY  £,A(s)  |,  £  >  0,  n  <  -  J,  |Jn+1  I d Y€(s)  |,  £  >  0,  n  <  »} 

is  uniformly  integrable.  The  first  set  is  uniformly  integrable  by  Theorem  8.3.  Let 
£  index  a  weakly  convergent  subsequence  (X£'A(),  Y£>A()}  and  {X£(  ),  Y£(  )}  with 
limit  pairs  (XA(  ),  YA(  ))  and  (X(  ),  Y(  )).  Then  XA(  )  is  the  (SAv(  ))  -  reflecting 
diffusion  and  YA(  )  *  YA(  ).  Thus,  by  the  weak  convergence 

8/i  +  li!!LV€(x)  J  i^Vo(x-  y€>  >  V0(x,  Y)  >  V0(x,  Y)  =  V(x), 

€  € 

A  __ 

Urn  V£(x.Y£,A)  =  V(x,Ya). 

Another  weak  convergence  argument  and  the  uniqueness  assumption  on 
the  reflecting  diffusion  X()  under  policy  Y()  yields  the  convergence  of 
(XA(  •  ),Ya(  • ))  to  (X(  • ),  Y(  •)).  Also,  the  set  {J^+1|dYA(s)l,  A  <  A0,  n  <  »}  is 
uniformly  integrable.  The  theorem  now  follows  by  choosing  A  small  enough  so 
that  Ya(  )  is  8/4  optimal  for  X(  )  and  using  the  optimality  of  Y(  )  for  X(  ). 


Q.E.D. 


Remarks  and  Extensions.  If  the  bounding  set  Sj  is  dropped,  then  we  might  assume 
that  the  optimal  control  is  a  (S,v(  ))-rcflecting  policy,  but  where  S  is  not  necessarily 
compact.  In  this  case,  given  6  >  0,  there  are  numbers  K&  and  6/4-  optimal  policies 
for  X£(  )  and  X(  )  for  which  dY€(t)  (resp.,  dY(t))  equals  zero  after  the  first  time 
that  the  variation  exceeds  Kg.  In  this  case,  we  have  the  required  uniform 
integrability  and  the  theorem  is  easier  to  prove. 


References 


[1]  J.P.  Lehoezky  and  S.E.  Shreve,  "Absolutely  Continuous  and  Singular 
Stochastic  Control",  STOCHASTICS,  17,  1986,  91-110. 

[2]  I.  Karatzas  and  S.E.  Shreve,  "Equivalent  Models  for  Finite  Fuel  Stochastic 
Control",  STOCHASTICS,  18,  1986,  245-276. 

[3]  I.  Karatzas,  "A  Class  of  Singular  Stochastic  Control  Problems”,  Adv.  Appl. 
Prob.,  15,  1983,  225-254. 

[4]  J.M.  Harrison,  Brownian  Motion  and  Stochastic  Flow  Systems ,  Wiley,  New 
York,  1985. 

[5]  J.M.  Harrison  and  M.I.  Taksar,  "Instantaneous  Control  of  Brownian  Motion”, 
Math.  Oper.  Res.,  8,  1983,  454-466. 

[6]  H.  Kushner  and  W.  Runggaldicr,  "Nearly  Optimal  State  Feedback  Controls 
for  Stochastic  Systems  with  Wideband  Noise  Disturbances",  to  appear  SIAM 
J.  on  Control  and  Optimization.  Also,  LCDS  Rept.  85-23  (1985)  Brown 
University. 

[7]  H.J.  Kushner  and  W.  Runggaldier,  "Filtering  and  Control  for  Wide  Bandwidth 
Noise  Driven  Systems",  LCDS  Rept.  86-8,  Jan.  1986,  to  appear  IEEE  Trans,  on 
Automatic  Control. 

[8]  H.J.  Kushner,  Approximation  and  Weak  Convergence  Methods  for  Random 
Processes;  with  Applications  to  Stochastic  Systems  Theory ,  MIT  Press, 
Cambridge,  MA,  USA,  1984. 

[9]  J.L.  Mcnaldi  and  M.  Robin,  "On  Some  Cheap  Control  Problems  for  Diffusion 
Processes",  Trans,  Amer.  Math.  Soc,  278,  1983,  771-802. 

[10]  C.  Dellachcrie  and  P.A.  Meyer,  Proababilities  el  Potenticl,  Hermann,  Paris; 
North  Holland,  Amsterdam. 

[11]  P.A.  Meyer  and  W.A.  Zheng,  "Tightness  Criteria  for  Laws  of 
Scmimartingalcs",  Ann.  Inst.  Henri  Poincare,  20,  1984,  353-372. 

[12]  G.  Blankenship  and  G.C.  Papanicolaou,  "Stability  and  Control  of  Stochastic 
Systems  with  Wide  Band  Noise  Disturbances",  SIAM  J.  Appl.  Math,  34, 
1978,  437-476. 

[13]  G.C.  Papanicolaou  and  W.  Kohler,  "Asymptotic  Theory  of  Mixing  Ordinary 
Differential  Equations",  Comm.  Pure.  Appl.  Math.,  27,  1974,  641-668. 

[14]  H.  Kushner,  "Jump  Diffusion  Approximations  for  Ordinary  Differential 
Equations  with  Wideband  Random  Right  Hand  Sides",  SIAM  J.  on  Control 
and  Optimization,  17,  1979,  729-744. 

[15]  J.L.  Doob,  Stochastic  Processes ,  1952,  Wiley,  New  York. 


